{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 16. Extracting Text from Web Pages\n",
    "\n",
    "## 16.1. The Structure of HTML Documents\n",
    "\n",
    "The `<html>` tag specifies the start on the entire HTM document. Lets utilize that tag to create a document composed of just a single word; _Hello_.\n",
    "\n",
    "**Listing 16. 1. Defining a simple HTML string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_contents = \"<html>Hello</html>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can render `html_contents` directly in an IPython Jupyter Notebook. We simply need to import `HTML` and `display` from `IPython.core.display`. Afterwards, executing `display(HTML(html_contents))` will display the rendered output. \n",
    "\n",
    "**Listing 16. 2. Rendering an HTML string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html>Hello</html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "def render(html_contents): display(HTML(html_contents))\n",
    "render(html_contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ve rendered our HTML document. It’s not very impressive. The body is composed of a single word. Furthermore, the document lacks a title. Let’s assign the document a title, using the `<title>` tag.  \n",
    "\n",
    "**Listing 16. 3. Defining an HTML title**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "title = \"<title>Data Science is Fun</title>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we’ll nest the title within `<html>` and `</html>` by running `html_contents = f\"<html>{title}Hello</html>\"`. \n",
    "\n",
    "**Listing 16. 4. Adding a title to the HTML string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html><title>Data Science is Fun</title>Hello</html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "html_contents = f\"<html>{title}Hello</html>\"\n",
    "render(html_contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The title reflects vital information despite its absence from the body of the document. This critical distinction is commonly emphasized using `<head>` and `<body>` tags. The content delimited by the HTML `<body>` tag will appear in the body of the output. Meanwhile, `<head>` delimits vital information that is not rendered within the body. \n",
    "\n",
    "**Listing 16. 5. Adding a head and body to the HTML string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "head = f\"<head>{title}</head>\"\n",
    "body = \"<body>Hello</body>\"\n",
    "html_contents = f\"<html> {title} {body}</html>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Occasionally, we’ll want to display a document’s title within the body of a page. This visualized title is referred is the page _header_. It is demarcated with the `<h1>` tag. \n",
    "\n",
    "**Listing 16. 6. Adding a header to the HTML string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html> <title>Data Science is Fun</title> <body><h1>Data Science is Fun</h1>Hello</body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "header = \"<h1>Data Science is Fun</h1>\"\n",
    "body =  f\"<body>{header}Hello</body>\"\n",
    "html_contents = f\"<html> {title} {body}</html>\"\n",
    "render(html_contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HTML documents usually contain multiple sentences, enclosed in multiple paragraphs.  such paragraphs are marked with a `<p>` tag.  Let’s add two consecutive paragraphs to our HTML. \n",
    "\n",
    "**Listing 16. 7. Adding paragraphs to the HTML string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html> <title>Data Science is Fun</title> <body><h1>Data Science is Fun</h1><p>Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 </p><p>Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 </p></body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "paragraphs = ''\n",
    "for i in range(2):\n",
    "    paragraph_string = f\"Paragraph {i} \" * 40\n",
    "    paragraphs += f\"<p>{paragraph_string}</p>\"\n",
    "    \n",
    "body = f\"<body>{header}{paragraphs}</body>\"\n",
    "html_contents = f\"<html> {title} {body}</html>\"\n",
    "render(html_contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can discriminate between `<p>` tags by assigning each tag a unique id. The id can be inserted directly into the tag brackets. The added _id_ is referred as an **attribute** of the paragraph element. Attributes are inserted into element start-tags in order to track useful tag information.\n",
    "\n",
    "**Listing 16. 8. Adding id attributes to the paragraphs**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "paragraphs = ''\n",
    "for i in range(2):\n",
    "    paragraph_string = f\"Paragraph {i} \" * 40\n",
    "    attribute = f\"id='paragraph {i}'\"\n",
    "    paragraphs += f\"<p {attribute}>{paragraph_string}</p>\"\n",
    "    \n",
    "body = f\"<body>{header}{paragraphs}</body>\"\n",
    "html_contents = f\"<html> {title} {body}</html>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The internet is built on-top of **hyperlinks**, which are clickable texts that connect web pages. Each hyperlink is marked by an anchor tag, `<a>`. Furthermore, the URL of the link is provided using the _href_ attribute. Below, we’ll create a hyperlink that reads _Data Science Bookcamp_. We’ll link that clickable text to the website for this book. \n",
    "\n",
    "**Listing 16. 9. Adding a hyperlink to the HTML string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html> <title>Data Science is Fun</title> <body><h1>Data Science is Fun</h1><p id='paragraph 0'>Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 </p><p id='paragraph 1'>Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 </p><p id='paragraph 2'>Here is a link to <a href='https://www.manning.com/books/data-science-bookcamp'>Data Science Bookcamp</a></p></body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "link_text = \"Data Science Bookcamp\"\n",
    "url = \"https://www.manning.com/books/data-science-bookcamp\"\n",
    "hyperlink = f\"<a href='{url}'>{link_text}</a>\"\n",
    "new_paragraph = f\"<p id='paragraph 2'>Here is a link to {hyperlink}</p>\"\n",
    "paragraphs += new_paragraph\n",
    "body = f\"<body>{header}{paragraphs}</body>\"\n",
    "html_contents = f\"<html> {title} {body}</html>\"\n",
    "render(html_contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beyond just headers and paragraphs, we can also visualize lists of texts within an HTML document. Suppose, for instance, that we wish to display a list of popular data science libraries. We’ll start by defining that list in Python.\n",
    "\n",
    "**Listing 16. 10. Defining a list of data science libraries**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "libraries = ['NumPy', 'Scipy', 'Pandas', 'Scikit-Learn']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we’ll demarcate every item in our list with an `<li>` tag, which stands for _list item_. \n",
    "\n",
    "**Listing 16. 11. Demarcating list items with an li tag.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "items = ''\n",
    "for library in libraries:\n",
    "    items += f\"<li>{library}</li>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we’ll nest the `items` string within a `<ul>` tag, where `ul` stands for unstructured list. Afterwards, we’ll append the unstructured list to the body our HTML. \n",
    "\n",
    "**Listing 16. 12. Adding an unstructured list to the HTML string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html> <title>Data Science is Fun</title> <body><h1>Data Science is Fun</h1><p id='paragraph 0'>Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 </p><p id='paragraph 1'>Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 </p><p id='paragraph 2'>Here is a link to <a href='https://www.manning.com/books/data-science-bookcamp'>Data Science Bookcamp</a></p><h2>Common Data Science Libraries</h2><ul><li>NumPy</li><li>Scipy</li><li>Pandas</li><li>Scikit-Learn</li></ul></body></html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "unstructured_list = f\"<ul>{items}</ul>\"\n",
    "header2 = '<h2>Common Data Science Libraries</h2>'\n",
    "body = f\"<body>{header}{paragraphs}{header2}{unstructured_list}</body>\"\n",
    "html_contents = f\"<html> {title} {body}</html>\"\n",
    "render(html_contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, it’s worth noting that our HTML body is divided into two distinct parts. Typically, such divisions are captured using special `<div>` tags. Usually, each `<div>` tag is distinguished by some attribute. If the attribute is unique to a division, then that attribute is an `id`. Otherwise, if the attribute is shared by more than one division, then a special `class` signifier is used. For consistency's sake, we’ll now divide our two sections by nesting them within two different divisions.\n",
    "\n",
    "**Listing 16. 13. Adding divisions to the HTML string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "div1 = f\"<div id='paragraphs' class='text'>{paragraphs}</div>\"\n",
    "div2 = f\"<div id='list' class='text'>{header2}{unstructured_list}</div>\"\n",
    "div3 = \"<div id='empty' class='empty'></div>\"\n",
    "body = f\"<body>{header}{div1}{div2}{div3}</body>\"\n",
    "html_contents = f\"<html> {title}{body}</html>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ve made many changes to our `html_contents` string. Lets actually review its altered contents.\n",
    "\n",
    "**Listing 16. 14. Printing the altered HTML string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html> <title>Data Science is Fun</title><body><h1>Data Science is Fun</h1><div id='paragraphs' class='text'><p id='paragraph 0'>Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 </p><p id='paragraph 1'>Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 </p><p id='paragraph 2'>Here is a link to <a href='https://www.manning.com/books/data-science-bookcamp'>Data Science Bookcamp</a></p></div><div id='list' class='text'><h2>Common Data Science Libraries</h2><ul><li>NumPy</li><li>Scipy</li><li>Pandas</li><li>Scikit-Learn</li></ul></div><div id='empty' class='empty'></div></body></html>\n"
     ]
    }
   ],
   "source": [
    "print(html_contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The printed output is a mess! The HTML contents are nearly unreadable. Also, extracting individual elements from `html_contents` is exceedingly difficult. \n",
    "\n",
    "**Listing 16. 15. Extracting the HTML title using basic Python**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Science is Fun\n"
     ]
    }
   ],
   "source": [
    "split_contents = html_contents.split('>')\n",
    "for i, substring in enumerate(split_contents):\n",
    "    if substring.endswith('<title'):\n",
    "        next_string = split_contents[i + 1]\n",
    "        title = next_string.split('<')[0]\n",
    "        print(title)\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is there a cleaner way to extract elements from HTML documents? Yes! We don’t need to manually parse the documents. Instead, we can just leverage the external Beautiful Soup library.\n",
    "\n",
    "## 16.2. Parsing HTML using Beautiful Soup\n",
    "\n",
    "Lets import a `BeautifulSoup` class from `bs4`. Following a common convention, we’ll import `BeautifulSoup` as simply `bs`.\n",
    "\n",
    "**Listing 16. 16. Importing the `BeautifulSoup` class**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup as bs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ll now initialize the `BeautifulSoup` class by running `bs(html_contents)`. In keeping with convention, we’ll assign the initialized object to a `soup` variable.\n",
    "\n",
    "**Listing 16. 17. Initializing `BeautifulSoup` using an HTML string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = bs(html_contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our `soup` object tracks all elements in the parsed HTML. We can output these elements in a clean, readable format by running the `soup.prettify()` method.\n",
    "\n",
    "**Listing 16. 18. Printing readable HTML with Beautiful Soup**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html>\n",
      " <title>\n",
      "  Data Science is Fun\n",
      " </title>\n",
      " <body>\n",
      "  <h1>\n",
      "   Data Science is Fun\n",
      "  </h1>\n",
      "  <div class=\"text\" id=\"paragraphs\">\n",
      "   <p id=\"paragraph 0\">\n",
      "    Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0\n",
      "   </p>\n",
      "   <p id=\"paragraph 1\">\n",
      "    Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1\n",
      "   </p>\n",
      "   <p id=\"paragraph 2\">\n",
      "    Here is a link to\n",
      "    <a href=\"https://www.manning.com/books/data-science-bookcamp\">\n",
      "     Data Science Bookcamp\n",
      "    </a>\n",
      "   </p>\n",
      "  </div>\n",
      "  <div class=\"text\" id=\"list\">\n",
      "   <h2>\n",
      "    Common Data Science Libraries\n",
      "   </h2>\n",
      "   <ul>\n",
      "    <li>\n",
      "     NumPy\n",
      "    </li>\n",
      "    <li>\n",
      "     Scipy\n",
      "    </li>\n",
      "    <li>\n",
      "     Pandas\n",
      "    </li>\n",
      "    <li>\n",
      "     Scikit-Learn\n",
      "    </li>\n",
      "   </ul>\n",
      "  </div>\n",
      "  <div class=\"empty\" id=\"empty\">\n",
      "  </div>\n",
      " </body>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we want to access an individual element, such as the title. The `soup` object provides that access through its `find` method. \n",
    "\n",
    "**Listing 16. 19. Extracting the title with Beautiful Soup**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<title>Data Science is Fun</title>\n"
     ]
    }
   ],
   "source": [
    "title = soup.find('title')\n",
    "print(title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The outputted `title` appears to be an HTML string that’s demarcated by the title tags. However, our `title` variable is not a string. Rather, it’s an initialized Beautiful Soup `Tag` class. \n",
    "\n",
    "**Listing 16. 20. Outputting the title’s data type**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'bs4.element.Tag'>\n"
     ]
    }
   ],
   "source": [
    "print(type(title))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each `Tag` object  contains a `text` attribute, which maps to the text within the tag. Thus, printing `title.text` will return  _Data Science is Fun_.\n",
    "\n",
    "**Listing 16. 21. Outputting the title’s text attribute**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Science is Fun\n"
     ]
    }
   ],
   "source": [
    "print(title.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ve accessed our `title` tag by running `soup.find('title')`. Additionally, we can access that same tag simply by running `soup.title`. Therefore, running `soup.title.text` will return a string that’s equal to `title.text`.\n",
    "\n",
    "**Listing 16. 22. Accessing the title’s text attribute from `soup`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert soup.title.text == title.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this same manner, we can access the body of our document by running `soup.body`.\n",
    "\n",
    "**Listing 16. 23. Accessing the body’s text attribute from soup**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Science is FunParagraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Here is a link to Data Science BookcampCommon Data Science LibrariesNumPyScipyPandasScikit-Learn\n"
     ]
    }
   ],
   "source": [
    "body = soup.body\n",
    "print(body.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our output is an aggregation of all the text within the body. It is virtually unreadable. Rather than outputting all the text, we should instead narrow the scope of our output. Lets print the text of just the first paragraph. \n",
    "\n",
    "**Listing 16. 24. Accessing the text of the first paragraph**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 \n"
     ]
    }
   ],
   "source": [
    "assert body.p.text == soup.p.text\n",
    "print(soup.p.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accessing `body.p` returns the first paragraph in `body`. How do we access the remaining two paragraphs? Well, we can utilize the `find_all` method. Running `body.find_all('p')` will return a list of all the paragraph tags within the body.\n",
    "\n",
    "**Listing 16. 25. Accessing all paragraphs in the body**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PARAGRAPH 0:\n",
      "Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 \n",
      "\n",
      "PARAGRAPH 1:\n",
      "Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 \n",
      "\n",
      "PARAGRAPH 2:\n",
      "Here is a link to Data Science Bookcamp\n"
     ]
    }
   ],
   "source": [
    "paragraphs = body.find_all('p')\n",
    "for i, paragraph in enumerate(paragraphs):\n",
    "    print(f\"\\nPARAGRAPH {i}:\")\n",
    "    print(paragraph.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we access our list of bullet-points by running `body.find_all('li')`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['NumPy', 'Scipy', 'Pandas', 'Scikit-Learn']\n"
     ]
    }
   ],
   "source": [
    "print([bullet.text for bullet\n",
    "       in  body.find_all('li')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we wish to access an element with a unique id of `x`. In order to search on that attribute id, we simply need to execute `find(id=x)`. With this in mind, lets output the text of the final paragraph, whose assigned id is `paragraph 2`.\n",
    "\n",
    "**Listing 16. 27. Accessing a paragraph by id**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here is a link to Data Science Bookcamp\n"
     ]
    }
   ],
   "source": [
    "paragraph_2 = soup.find(id='paragraph 2')\n",
    "print(paragraph_2.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The contents of `paragraph_2` include a web link to _Data Science Bookcamp_. The actual url is stored within a `href` attribute. Beautiful Soup permits us to access any attribute using the `get` method. Thus, running `paragraph_2.get(id)` will return _paragraph 2_. Subsequently, running `paragraph_2.a.get(href)` will return the url. \n",
    "\n",
    "**Listing 16. 28. Accessing an attribute within a tag**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.manning.com/books/data-science-bookcamp\n"
     ]
    }
   ],
   "source": [
    "assert paragraph_2.get('id') == 'paragraph 2'\n",
    "print(paragraph_2.a.get('href'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Not all our attributes are unique. For instance, two of our three division elements share the `class` attribute of _text_. How do we obtain just those two divisions where the class is set to _text_? Well, we just simply need to run `body.find_all('div', class_='text')`. \n",
    " \n",
    "**Listing 16. 29. Accessing divisions by their shared class attribute**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Division with id 'paragraphs':\n",
      "Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 0 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Paragraph 1 Here is a link to Data Science Bookcamp\n",
      "\n",
      "Division with id 'list':\n",
      "Common Data Science LibrariesNumPyScipyPandasScikit-Learn\n"
     ]
    }
   ],
   "source": [
    "for division in soup.find_all('div', class_='text'):\n",
    "    id_ = division.get('id')\n",
    "    print(f\"\\nDivision with id '{id_}':\")\n",
    "    print(division.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Beautiful Soup library also allows us to edit individual elements. For example, given a `tag` object, we can delete that object by running `tag.decompose()`. Lets delete the first two paragraphs.\n",
    "\n",
    "**Listing 16. 30. Paragraph deletion with Beautiful Soup**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here is a link to Data Science Bookcamp\n"
     ]
    }
   ],
   "source": [
    "body.find(id='paragraph 0').decompose()\n",
    "soup.find(id='paragraph 1').decompose()\n",
    "print(body.find(id='paragraphs').text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, we’re able to insert new tags into the HTML. Suppose we wish to insert a new paragraph into our final empty division. To do so, we must first create a new paragraph element. \n",
    "\n",
    "**Listing 16. 31. Initializing an empty paragraph `Tag`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<p></p>\n"
     ]
    }
   ],
   "source": [
    "new_paragraph = soup.new_tag('p')\n",
    "print(new_paragraph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we must update the initialized paragraph’s text, by assigning it to `new_paragraph.string`. \n",
    "\n",
    "**Listing 16. 32. Updating the text of an empty paragraph**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<p>This paragraph is new</p>\n"
     ]
    }
   ],
   "source": [
    "new_paragraph.string = \"This paragraph is new\"\n",
    "print(new_paragraph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we must append the updated `new_paragraph`  to an existing `Tag` object. Given two `Tag` objects; `tag1` and `tag`, we can insert `tag1` into `tag2` by running `tag2.append(tag1)`. Thus, running `soup.find(id='empty').append(new_paragraph)` should append to paragraph to the empty division.\n",
    "\n",
    "**Listing 16. 33. Paragraph insertion with Beautiful Soup**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<html>\n",
       " <title>\n",
       "  Data Science is Fun\n",
       " </title>\n",
       " <body>\n",
       "  <h1>\n",
       "   Data Science is Fun\n",
       "  </h1>\n",
       "  <div class=\"text\" id=\"paragraphs\">\n",
       "   <p id=\"paragraph 2\">\n",
       "    Here is a link to\n",
       "    <a href=\"https://www.manning.com/books/data-science-bookcamp\">\n",
       "     Data Science Bookcamp\n",
       "    </a>\n",
       "   </p>\n",
       "  </div>\n",
       "  <div class=\"text\" id=\"list\">\n",
       "   <h2>\n",
       "    Common Data Science Libraries\n",
       "   </h2>\n",
       "   <ul>\n",
       "    <li>\n",
       "     NumPy\n",
       "    </li>\n",
       "    <li>\n",
       "     Scipy\n",
       "    </li>\n",
       "    <li>\n",
       "     Pandas\n",
       "    </li>\n",
       "    <li>\n",
       "     Scikit-Learn\n",
       "    </li>\n",
       "   </ul>\n",
       "  </div>\n",
       "  <div class=\"empty\" id=\"empty\">\n",
       "   <p>\n",
       "    This paragraph is new\n",
       "   </p>\n",
       "  </div>\n",
       " </body>\n",
       "</html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "soup.find(id='empty').append(new_paragraph)\n",
    "render(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 16.3. Downloading and Parsing Online Data \n",
    "\n",
    "The Beautiful Soup library allows to easily parse, analyze, and edit HTML documents. In most cases, these documents must be downloaded directly from the web. Lets briefly review the procedure for downloading HTML files. We’ll start by importing the `urlopen` function.\n",
    "\n",
    "**Listing 16. 34. Importing the `urlopen` function**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlopen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the url of an online document, we can download the associated HTML contents by running `urlopen(url).read()`. Below, we’ll use `urlopen` to download the Manning website for this book. Afterwards, we’ll print the first 1000 characters of the downloaded HTML.\n",
    "\n",
    "**Listing 16. 35. Downloading an HTML document**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'\\n<!DOCTYPE html>\\n<!--[if lt IE 7 ]> <html lang=\"en\" class=\"no-js ie6 ie\"> <![endif]-->\\n<!--[if IE 7 ]>    <html lang=\"en\" class=\"no-js ie7 ie\"> <![endif]-->\\n<!--[if IE 8 ]>    <html lang=\"en\" class=\"no-js ie8 ie\"> <![endif]-->\\n<!--[if IE 9 ]>    <html lang=\"en\" class=\"no-js ie9 ie\"> <![endif]-->\\n<!--[if (gt IE 9)|!(IE)]><!--> <html lang=\"en\" class=\"no-js\"><!--<![endif]-->\\n\\n<head>\\n    <title>Manning | Data Science Bookcamp</title>\\n    \\n\\n\\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\">\\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\">\\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1, maximum-scale=1, user-scalable=0\">\\n<meta name=\"application-name\" content=\"Data Science Bookcamp\"/>\\n<meta name=\"apple-mobile-web-app-title\" content=\"Data Science Bookcamp\"/>\\n\\n<meta property=\"og:title\" content=\"Data Science Bookcamp\"/>\\n<meta name=\"twitter:title\" content=\"Data Science Bookcamp\"/>\\n\\n<meta name=\"twitter:site\" content=\"&#64;manningbooks\"/>\\n<meta name=\"twitt'\n"
     ]
    }
   ],
   "source": [
    "url = \"https://www.manning.com/books/data-science-bookcamp\"\n",
    "html_contents = urlopen(url).read()\n",
    "print(html_contents[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets extract the title from our messy HTML using Beautiful Soup.\n",
    "\n",
    "**Listing 16. 36. Accessing the title with Beautiful Soup**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Manning | Data Science Bookcamp\n"
     ]
    }
   ],
   "source": [
    "soup = bs(html_contents)\n",
    "print(soup.title.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Leveraging our `soup` object, we can further analyze the page. For instance, we can extract the division that contains an _about the book_ header in order to print a description of this book.\n",
    "\n",
    "**Listing 16. 37. Accessing a description of this book**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "about the book\n",
      "\n",
      "Data Science Bookcamp is a comprehensive set of challenging projects carefully designed to grow your data science skills from novice to master. Veteran data scientist Leonard Apeltsin sets five increasingly difficult exercises that test your abilities against the kind of problems you’d encounter in the real world. As you solve each challenge, you’ll acquire and expand the data science and Python skills you’ll use as a professional data scientist. Ranging from text processing to machine learning, each project comes complete with a unique downloadable data set and a fully-explained step-by-step solution. Because these projects come from Dr. Apeltsin’s vast experience, each solution highlights the most likely failure points along with practical advice for getting past unexpected pitfalls.  When you wrap up these five awesome exercises, you’ll have a diverse relevant skill set that’s transferable to working in industry.\n",
      "    \n"
     ]
    }
   ],
   "source": [
    "for division in soup.find_all('div'):\n",
    "    header = division.h2\n",
    "    if header is None:\n",
    "        continue\n",
    "        \n",
    "    if header.text.lower() == 'about the book':\n",
    "        print(division.text)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
